{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SzvDfdQNnS4e",
    "origin_pos": 0
   },
   "source": [
    "# Encoder-Decoder Architecture\n",
    ":label:`sec_encoder-decoder`\n",
    "\n",
    "As we have discussed in \n",
    ":numref:`sec_machine_translation`,\n",
    "machine translation\n",
    "is a major problem domain for sequence transduction models,\n",
    "whose input and output are\n",
    "both variable-length sequences.\n",
    "To handle this type of inputs and outputs,\n",
    "we can design an architecture with two major components.\n",
    "The first component is an *encoder*:\n",
    "it takes a variable-length sequence as the input and transforms it into a state with a fixed shape.\n",
    "The second component is a *decoder*:\n",
    "it maps the encoded state of a fixed shape\n",
    "to a variable-length sequence.\n",
    "This is called an *encoder-decoder* architecture,\n",
    "which is depicted in :numref:`fig_encoder_decoder`.\n",
    "\n",
    "![The encoder-decoder architecture.](https://github.com/d2l-ai/d2l-en-colab/blob/master/img/encoder-decoder.svg?raw=1)\n",
    ":label:`fig_encoder_decoder`\n",
    "\n",
    "Let us take machine translation from English to French\n",
    "as an example.\n",
    "Given an input sequence in English:\n",
    "\"They\", \"are\", \"watching\", \".\",\n",
    "this encoder-decoder architecture\n",
    "first encodes the variable-length input into a state,\n",
    "then decodes the state \n",
    "to generate the translated sequence token by token\n",
    "as the output:\n",
    "\"Ils\", \"regardent\", \".\".\n",
    "Since the encoder-decoder architecture\n",
    "forms the basis\n",
    "of different sequence transduction models\n",
    "in subsequent sections,\n",
    "this section will convert this architecture\n",
    "into an interface that will be implemented later.\n",
    "\n",
    "## Encoder\n",
    "\n",
    "In the encoder interface,\n",
    "we just specify that\n",
    "the encoder takes variable-length sequences as the input.\n",
    "The implementation will be provided \n",
    "by any model that inherits this base `Encoder` class.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load ../utils/djl-imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "public abstract class Encoder extends AbstractBlock {\n",
    "\n",
    "    /* The base encoder interface for the encoder-decoder architecture. */\n",
    "    private static final byte VERSION = 1;\n",
    "\n",
    "    public Encoder() {\n",
    "        super(VERSION);\n",
    "    }\n",
    "\n",
    "    @Override\n",
    "    abstract protected NDList forwardInternal(\n",
    "            ParameterStore parameterStore,\n",
    "            NDList inputs,\n",
    "            boolean training,\n",
    "            PairList<String, Object> params);\n",
    "\n",
    "    @Override\n",
    "    public Shape[] getOutputShapes(Shape[] inputShapes) {\n",
    "        throw new UnsupportedOperationException(\"Not implemented\");\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hEmJQeipnS4k",
    "origin_pos": 3
   },
   "source": [
    "## Decoder\n",
    "\n",
    "In the following decoder interface,\n",
    "we add an additional `initState` function\n",
    "to convert the encoder output (`encOutputs`)\n",
    "into the encoded state.\n",
    "Note that this step\n",
    "may need extra inputs such as \n",
    "the valid length of the input,\n",
    "which was explained\n",
    "in :numref:`subsec_mt_data_loading`.\n",
    "To generate a variable-length sequence token by token,\n",
    "every time the decoder\n",
    "may map an input (e.g., the generated token at the previous time step)\n",
    "and the encoded state\n",
    "into an output token at the current time step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "public abstract class Decoder extends AbstractBlock {\n",
    "\n",
    "    /* The base decoder interface for the encoder-decoder architecture. */\n",
    "    private static final byte VERSION = 1;\n",
    "\n",
    "    public NDArray attentionWeights;\n",
    "\n",
    "    public Decoder() {\n",
    "        super(VERSION);\n",
    "    }\n",
    "\n",
    "    @Override\n",
    "    abstract protected NDList forwardInternal(\n",
    "            ParameterStore parameterStore,\n",
    "            NDList inputs,\n",
    "            boolean training,\n",
    "            PairList<String, Object> params);\n",
    "\n",
    "    abstract public NDList initState(NDList encOutputs);\n",
    "\n",
    "    @Override\n",
    "    public Shape[] getOutputShapes(Shape[] inputShapes) {\n",
    "        throw new UnsupportedOperationException(\"Not implemented\");\n",
    "    }\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FbTu97BYnS4l",
    "origin_pos": 6
   },
   "source": [
    "## Putting the Encoder and Decoder Together\n",
    "\n",
    "In the end,\n",
    "the encoder-decoder architecture\n",
    "contains both an encoder and a decoder,\n",
    "with optionally extra arguments.\n",
    "In the forward propagation,\n",
    "the output of the encoder\n",
    "is used to produce the encoded state,\n",
    "and this state\n",
    "will be further used by the decoder as one of its input.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "public class EncoderDecoder extends AbstractBlock {\n",
    "\n",
    "    /* The base class for the encoder-decoder architecture. */\n",
    "    private static final byte VERSION = 1;\n",
    "\n",
    "    public Encoder encoder;\n",
    "    public Decoder decoder;\n",
    "\n",
    "    public EncoderDecoder(Encoder encoder, Decoder decoder) {\n",
    "        super(VERSION);\n",
    "\n",
    "        this.encoder = encoder;\n",
    "        this.addChildBlock(\"encoder\", this.encoder);\n",
    "        this.decoder = decoder;\n",
    "        this.addChildBlock(\"decoder\", this.decoder);\n",
    "    }\n",
    "\n",
    "    /** {@inheritDoc} */\n",
    "    @Override\n",
    "    public void initializeChildBlocks(NDManager manager, DataType dataType, Shape... inputShapes) {\n",
    "    }\n",
    "\n",
    "    @Override\n",
    "    protected NDList forwardInternal(ParameterStore parameterStore, NDList inputs, boolean training, PairList<String, Object> params) {\n",
    "        NDArray encX = inputs.get(0);\n",
    "        NDArray decX = inputs.get(1);\n",
    "        NDList encOutputs = this.encoder.forward(parameterStore, new NDList(encX), training, params);\n",
    "        NDList decState = this.decoder.initState(encOutputs);\n",
    "        return this.decoder.forward(parameterStore, new NDList(decX).addAll(decState), training, params);\n",
    "    }\n",
    "\n",
    "    @Override\n",
    "    public Shape[] getOutputShapes(Shape[] inputShapes) {\n",
    "        throw new UnsupportedOperationException(\"Not implemented\");\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "THFYJ0oJnS4l",
    "origin_pos": 9
   },
   "source": [
    "The term \"state\" in the encoder-decoder architecture\n",
    "has probably inspired you to implement this\n",
    "architecture using neural networks with states.\n",
    "In the next section,\n",
    "we will see how to apply RNNs to design \n",
    "sequence transduction models based on \n",
    "this encoder-decoder architecture.\n",
    "\n",
    "\n",
    "## Summary\n",
    "\n",
    "* The encoder-decoder architecture can handle inputs and outputs that are both variable-length sequences, thus is suitable for sequence transduction problems such as machine translation.\n",
    "* The encoder takes a variable-length sequence as the input and transforms it into a state with a fixed shape.\n",
    "* The decoder maps the encoded state of a fixed shape to a variable-length sequence.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Suppose that we use neural networks to implement the encoder-decoder architecture. Do the encoder and the decoder have to be the same type of neural network?  \n",
    "1. Besides machine translation, can you think of another application where the encoder-decoder architecture can be applied?\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "encoder-decoder.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Java",
   "language": "java",
   "name": "java"
  },
  "language_info": {
   "codemirror_mode": "java",
   "file_extension": ".jshell",
   "mimetype": "text/x-java-source",
   "name": "Java",
   "pygments_lexer": "java",
   "version": "14.0.2+12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
